Regarding test failures, it's an incompatibility with Scrapy 2.13 which I haven't thought of checking, either adding install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") to e.g. conftest.py will be enough or the tests need to be changed to actually call and await crawler.crawl() (I haven't looked at the code).

And it should be unrelated to the changes of this PR.

I added a conftest.py to make it pass the tests locally. The file content is:

import os

os.environ.setdefault(
    "TWISTED_REACTOR",
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
)

try:
    from twisted.internet import asyncioreactor
    asyncioreactor.install()
except Exception:
    pass

Should I add this change to the PR?

Yes please.

Though I don't think touching the TWISTED_REACTOR envvar makes sense, Scrapy should only operate on settings (I doubt it even reads this envvar, though I won't be surprised that it does via some deprecated mechanism).

Codecov Report

❌ Patch coverage is 76.47059% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.45%. Comparing base (63b3991) to head (784bf7e).

@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   85.71%   83.45%   -2.27%     
==========================================
  Files           4        5       +1     
  Lines         105      139      +34     
  Branches       19       25       +6     
==========================================
+ Hits           90      116      +26     
- Misses         11       16       +5     
- Partials        4        7       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.